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ABSTRACT 

The purposes of this study were to examine strengths 
and weaknesses of research methods used in reports of follpw-up 
studies in higher education and to develop a set of guidelines for 
conducting foilow-up studies in higher education. A careful 
examination of the literature was conducted to identify foilpw-up 
studies in higher education and 47 states were located. G3iese studies 
were critiqued in relation to sound research methods. Guidelines for 
conducting follow-up studies in higher education were also reported. . 
References are included. (Author) 
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Introduction 

The emerging discipline of institutional research is 
receiving noticeable attention through reports published 
in journals, dissertations, ERIC documents and the like. 
With the increased awareness of the discipline and the 
improved communication among institutional research- 
ers through various means of publications, not only is 
it timely to assess the state of the art of institutional 
research methodology, but also, to place some judgment 
concerning the quality of research methods used in con- 
ducting institutional research. An assessment of methods 
used in institutional research at this early stage of devel- 
opment may well prove worthwhile in strengthening 
the state of the art and in establishing common bases 
for comparability and replication of important studies 
in higher education. Such a consequence would also 
enhance the institutional research reports to adminis- 
trators for decision-making purposes. 

Problem 

This report is based upon the assumption that 
most institutional research studies are conducted in 
a provincial manner directed toward providing infor- 
mation delimited to a particular institution. This 
procedure in effect restricts inferences made and the 
abilily for comparable institutions to replicate similar 
studies and expand their frame of inference. The 
problem which exists, therefore, is that established 
research methods such as those used in the behavioral 
sciences have not been widely applied to the problems 
in institutional research. 

Objectives 

The field of institutional research extends to all 
phases of activities associated with institutions of 
higher education. Traditionally, components of in- 
stitutional research '*jave centered around students, 
faculty, space, fiscal matters, and physical facilities. 



The area of concern in this report relates to student 
studies-and specifically follow-up studies in higher 
education. Specific objectives for conducting the 
analysis are: 1) to examine strengths and weaknesses 
of research methods used in reports on follow-up 
studies in higher education and 2) to develop a set 
of guidelines for conducting follow-up studies by 
institutional researchers. 

Procedures 

Selection of Reports on Follow-Up Studies in 
Higher Education. 

A careful examination of the Education Index, 
ERIC publications from July, 1964 to date, and card 
catalogue files revealed 95 reports with titles relating 
to the general category "Follow-Up Studies in Higher 
Education." These 95 reports ^vere located in 62 
journal articles, 14 dissertations, and 19 ERIC docu- 
ments. Eighteen of the references could not be 
located . After examining those reports available it 
was revealed that only 47 of the reports were actually 
follow-up studies in higher education. The 30 reports 
which were not directly related to follow-up studies in 
higher education were concerned with literature 
surveys, with studies dealing with high school stu- 
dents, or with surveys of enrollees who were in active 
student status. All of these were excluded from the 
analysis of this report. 

Criteria Used for Analysis 

The criteria used to judge the reports of follow- 
up studies in iiigher education were obtained from 
Strauss (1969). Others have presented guidelines for 
evaluating educational research reports (Bixler, 1928; 
Dvorak, 1956; Farquharand Krumboltz, 1959; Johnson, 
1957; Stephens, 1967; Symonds, 1956;Wandt and 



others, 1967). Strauss, however, presented 20 criteria 
which'can be applied very easily to judge reports on 
educational research. The 20 criteria presented by 
Strauss do not give a "best fit" to institutional research 
reports, but the 20 criteria were used here because they 
can easily be adapted to institutional research reports 
without losing sight of the need for better methods in 
conducting institutional research studies. The 20 
criteria suggested by Strauss for evaluating educational 
research reports are: problem raised, previous work 
cited, objectives stated, hypotheses formulated, assum- 
ption's nnade, population studied, sample drawn, 
instruments used, design examined, procedure followed, 
safeguards taken, observations recorded, findings assem- 
bled, statistics interpreted, interpretations discussed, 
conclusions reached, limitations recognized, further 
work projected, improvements suggested, and clarity 
of report. The author of these guidelines, Strauss, 
recognized full well that the criteria were developed 
to evaluate experimental research reports in education. 

The analytical procedures used in this report are 
limited to the follow-up studies identified for analysis 
and the 20 criteria judiciously chosen as a framework 
for judgement of the reports. » 

The actual procedures for^aluating the research 
methods reported in the selectW follow-up studies in 
higher education consisted of dbreful comparison of the 
studies with the 20 criteria and recording whether or 
not each report met the criteria. To some degree the 
accuracy and completeness of the information reported 
in each study was judged. 

Findings 

Applying Strauss's 20 criteria for evaluating edu- 
cational research methods to selected reports of follow- 
up studies in higher education the following judgments 
were reached by this writer: 

1. Nearly all (91%) of the follow-up studies stated 
a problem in their reports; 9% did not have a 
statement of the problem. Of the 91% of the 
studies with statennents of the problem, nearly 
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half (46%) of the statements were clearly identi- 
fied, but 47% of the problem statements had to 
be searche^for. 

2. Citations of previous work in the follow-up 
studies were well formulated and related to the 
current problem in 22% of the reports; 48% gave 
very brief citations of previous work; 1 2% gave 
brief but improper citations; and 18% of the 
follow-up studies cited no previous work. 

3. Nearly three-fourths (72%) of the follow-up 
studies contained specific objectives, while 
28% of the reports contained no statement of 
objectives. 

4. Only a few of the follow-up studies were con- 
structed in a manner to test research hypo- 
theses (9%). 

5. Over one-half (64%) of the follow-up studies 
made assumptions concer.iing their research, 
and 36% reported assumptions which identi- 
fied samples, instruments, and the like which 
could not be controlled. 

6. Nearly all of the follow-up studies (99%) de- 
scribed the population used in their studies. 

7. Nearly one-fourth of the studies (24%) chose 
samples to use in follow-up (9% random, 15% 
other), while 76% chose to follow up the entire 
population. 

8. Nearly three-fourths (74%) of the follow-up 
studies used questionnaires to collect data. Of 
these, 85% described the questionnaire in detail, 
the other 15% only mentioned that a question- 
naire was used. Other means for collecting data 
included personal interviews (6%), records and 
transcripts (6%), standardized tests (3%), and 

1 1 % of the reports did not mention how the 
data were collected in their follow-up studies. 

9. Description of the statistical design used in 
the follow-up studies was identified in only 
21% of the reports; 79% did not indicate how 
the follow-up data were to be treated. 

10. Nearly one-half (44%) of the reports indicated 
that clear and logical procedures were to be 
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followed; 26% of the reports indicated procedures 
which could have been better organized; 15% of the 
reports indicated incomplete procedures; and 15% 
of the reports did not indicate any procedures to be 
followed. 

11. Over three-fourths (76%) of the follow-up studies 
did not report any safeguards taken to control 
errors, while 18% used statistical methods to con- 
trol bias errors, and 6% used sampling techniques 
to control bias which could enter into the find- 
ings of the follow-up study. 

12. Only one-four^* '24%) of the follow-up studies 
reported primary sources of data or test scores, 
ratings, replies to questionnaires used, etc., and 
76% of the reports did not report any primary 
sources of replies or responses. 

13. Nearly all of ihe reports (97%) presented the 
findings of the follow-up studies in cirar and 
well organized tables; only 3% of the reports 
lacked a clear presentation of the findings of 
their follow-up. 

14. The types of statistics reported in the follow- 
up studies were primarily per cents (52%) and 
sum of responses (36%). Other types of statis- 
tics reported included ratios, range, chi-square, 
correlations, and analysis of covariance statistics. 

15. Over one-half (61%) of the follow-up studies pre- 
sented complete and accurate interpretations of 
their findings; 9% presented inaccurate inferences; 
and 30% gave only limited interpretations of their 
findings. 

16. Almost all of the follow-up studies (90%) came 
to some type of conclusions, while 10% did not 
conclude anything from their study. 

17. About one-half (51%) of the follow-up studies 
recognized limitations of thair reports, while 
49% did not recognize any limitations to their 
study. 

18. Over one-third (36%) of the reports projected 
the need for fuither study, while J4% did not 
project any need for further study of their 
problem. 
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19. Over three-fourths (88%) of the follow-up studies 
did not suggest any means for improvement, 
while 12% of the reports gave suggestions on how 
to improve their projects or the reporting proce- 
dures. 

20. The follow-up studiss were grouped into two 
categories in regard to the clarity of the report: 
43% of the studies were rated good to excellent, 
and 57% were rated fair to poor in clarity. 

Discussion 

The first point which should be revealed at this time 
is simply that Strauss' 20 criteria for evaluating education- 
al research studies can be adapted to assess reports on fol- 
low-up studies in higher education, and as conducted in 
an institutional research context. Even though the nature 
of institutional research, as currently practiced, does not 
require the sophistication of research methods necessary 
for conducting experimental studies in education, most 
of these criteria were meaningful to the ordinary follow- 
up studies reported in the literature. The findings of 
this report, also, indicated that follow-up studies in high- 
er education can be improved if certain guidelines are 
followed. The strengths of research methods revealed in 
the reports on follow-up studies in higher education 
appeared to be: 1) statement of the problem, 2) previous 
work cited, 3) statement of objectives, 4) description 
and selection of the population, 5) assembling the find- 
ings, and 6) conclusions reached. Those criteria which 
could have been improved in the reports were: Dassump- 
tions under which the follow-up was conducted, 2) field 
testing and validating of instruments used, 3) clarifica- 
tion of design used, 4) complete description of proce- 
dures followed, 5) safeguards taken to control error or 
bias, 6) recording of primary sources of data, 7) more 
refined statistics for strong and accurate inferences, 

8) complete interpretations of findings in relation to the 
problems and objectives for conducting the follow-up, 

9) recognition of limitations of the study, 10) construc- 
tive criticism of the study for further improvements, 
and 1 1 ) clearness of the writing and sound organization 
of the report. 



The nature of a follow-up study in an institutional 
research setting and for meeting dissertation requirements 
are different |n an institutional research setting the need 
to conduct follow-uD studies has not been comprehensive 
enough to require rigid experimental research methods. 
iJsually, the need to conduct a follow-up study by Institu- 
tional researchers has been to describe where graduates of 
a single institution have gone and offer simple descrip- 
tions of selected characteristics of these graduates after 
taking a degree. Also, it is more desirable to use the en- 
tire population in such a setting than to select a random 
sample. Academic deans usually desire to know where 
all of their graduates have gone and what they are doing 
in contrast to a selected sample. 

For institutional researchers who are desirous of 
improving their research skills to strengthen approaches 
in conducting follow-up studies, most large campuses 
contain many courses on statistics, research design, 
psychometric testing, and the like. The reports analyzed 
in this study, however, indicate that only a knowledge 
of descriptive statistics has been used in the past in con- 
ducting follow-up studies. As the discipline of institu- 
tional research matures it will be desirable to strengthen 
the inferential base of institutional data and this wiM 
require a higher level of knowledge and research skills. 
Finally, it is only a matter of time before Journal 
of Institutional Research will be created. In order to 
support such a competitive journal it will be necessary 
to provide high quality reports. 

From the findings of this report the writer would 
suggest that the following guidelines can be used to 
strengthen the conducting and reporting of follow-up 
studies in higher education. The guidelines should be 
used in preparing or structuring the follow-up study as 
well as in reporting purposes. 



Guidelines for Conducting Follow-Up Studies in 
Higher Education 



1. Statement of the Problem. A problem may be stated 
in several different ways. A very desirable manner is 
to state the problem in question form which clearly 
explains wh^t you are trying to determine. The ques- 
tion should be stated in clear and precise terms so that 
the remainder of the study will have a logical flow or 
syntax. 

2. Justification for the Study. Why is the problem one 
of significance? Evidence from previous studies and 
recommendations should be documented to support 
the need to scudy the problem as stated. 

3. Objectivf s of the Study. The objectives should state 
the in^^cnt for doing the study and for attacking the 
problem. The statement of objectives, which originate 
from the problem, should be stated in clear, concise 
terms. Since follow-up studies are applied research 

by nature, it is infrequent that hypotheses ' re neces- 
sary to be statrd; and, therefore, objectives may be 
divided into a primary or general level with specific 
or secondary sub-levels. 

4. Delimitat.ons of the Study. Follow-up studies have 
been delimited by nature of populations surveyed, 
questionnaires used, per cent of leturns, type of 
data recorded, and the like. Because of the lack of 
very tight research controls, it is desirable to explain 
all the elements in the study which may bias the 
findings and restrict inferences of the interpretations 
and conclusions-such as validation of the instruments, 
population defined, sampling procedures, and statisti- 
cal designs used. All of these should be made clear. 

5. Procedures. The procedures for conducting a follow- 
up study in higher education should be described 
completely in order that others may replicate the 
study or sxtend your research. These procedures 
should answer the question, "How did you do it?" 
and usually consist of: 
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a Population Defined. The population should be 
described in complete detail-size, year of graduation, 
type sex, and other basic characteristics. 

b. Sample Selected. If a sample was selected froni 
the population, how was it selected? Random, strati- 
fied-random, cluster, incidental, etc. The characteris- 
tics of the sample should also be completely defined. 

c. Data Collection and Measuren)ent The tech- 
niques of how the data were collected and types of 
measurements recorded have usually received the 
most attention in conducting follow-up studies. Yet, 
very few persons have reported any field testing or 
validation attempts concerning questionnaires and 
interview techniques used. Relaxed approaches to 
gathering data by means of invalidated techniques 
results in data collected with inconsistencies and 
highly restrictive interpretations and inferences. An 
original questionnaire or interview technique should 
be developed upon sound theories and practices in 
psychometrics and field tested to the extent that 
these means will serve effectively to obtain desired 
information. Procedures involved to produce a valid 
questionnaire, interview technique, or any other 
means for data collection should be explained in 
detail. The nature of the data to be collected and 
measures recorded should be included in the descrip- 
tion of the methods used in collecting data. 

Occasionally, standardized tests are used m follow- 
up studies in higher education. The use of these tests 
should be justified, described, and properly acknovyledged. 

Methods for increasing the number of responses in 
follow-up studies in higher education is very important 
to describe. Some persons have experienced success 
in increasing the number of responses by additional 
mailings, phono calls, telegrams, and personal visits to 
the respondent's community. 

d. Plan of Data Analysis. The plan of data analysis 
should be related to the statement of the problem and 
objectives. The statistical design should reveal the form 



in which the data are recorded and to be analyzed. 
Simple and robust designs are preferable to elaborate 
designs which attempt too much. 

e. Report of Findings, The primary sources of 
data, such as replies to questionnaires, interviews, 
test scores, etc., should be recorded, these data 
sources would fit well into tables or appendices 
and provide other institutional researchers an 
opportunity to compare data between institutions. 

The findings should also be summari^ed in table 
form, graphs, charts or the like. These presentations 
should be complete, well-defined, and self-explanatory. 

f. Interpretation of the Findings. The findings 
should be interpreted in relation to the problem and 
procedures used in the study. One might ask, "Is 
the relationship between the problem and procedures 
clear and consistent, so that the findings are valid?" 
An occasional inferential error identified in follow- 
up studies is to obtain a 50 per cent return from a 
population or sample and make inferences about 
100 per cent of the population. Another short- 
coming identified occasionally in reports on follow- 
up studies was to make judgments regarding signifi- 
cant differences without appropriate statistical 
designs. It is desirable to obtain all meaning possible 
from the findings within a useful reference for insti- 
tutional decision-making and planning. 

g. Conclusions of the Study. The conclusions of 
the study should be consistent with the findings or 
obtained results and in relation to the problem and 
objectives of the study. A serious problem arises when 
conclusions of the report are not warranted upon the 
bases of the findings. 

h. Limitations Recognized. Every follow-up study 
in higher education will have limitations or weaknesses 
which may or may not be due to the investigator or 
unexpected events. Oftentimes unexpected events 
occur with the per cent of responses, measurements 
used, design used, population, and the like. These 
limitations should be revealed so that the consumer will 
be aware. 
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i. Recommendations for Further Study. A well 
planned and executed study will usually arouse 
interest and challenge further study. Comments to 
improve further studies should described in construc- 
tive criticisms. 
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